Girsanov Based Direct Policy Gradient Methods

Despite the plethora of reinforcement learning algorithms in machine learning and control, the majority of the work in this area relies on discrete time formulations of stochastic dynamics. In this work we present a new policy gradient algorithm for reinforcement learning in continuous state action spaces and continuous time. The derivation is based on successive application of Girsanov’s theorem and the use of the Radon Nikodým derivative as formulated for markov diffusion processes. The resulting policy gradient is reward weighted with the reward taking the form of a path integral. We apply the resulting algorithm in two simple examples for learning attractor landscapes in rhythmic and discrete movements.